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This publication is the result of an effort by the Nebraska State 
FOREWORD Department of Education to raise the level of management skills of 
Department personnel. The State Education Agency designated a section 
for planning and evaluation within the Division of Instruction. Planning and evaluation 
functions were specifically requested of the section by both Federal and State education 
programs. 

The Planning and Evaluation Proposal states the goal of this project as follows: "To prepare 
training materials and training opportunities for educational middle-management personnel 
in the process of evaluation." 

Twelve hours were specifically provided and used for the training sessions in evaluation 
skills for program administrators of the Instructional Division. This publication includes the 
materials supplied to the participants in a notebook for the sessions. It also includes the 
instruments that were used to collect information from the participants. The process and 
evaluation of these training sessions appear in a companion piece to this publication.lt is 
anticipated that divisions in other Departments of Education, as well as local education 
agencies, might be able to profit from the process and materials used in Nebraska for 
training middle-management personnel. 

The establishment of evaluation skills as an operational function among program 
administrators in the Division has been a sincere effort to raise the level of accountability 
toward that which is required in the investment of Federal, State, and local dollars. We 
believe that these sessions have contributed to that elfort. 



Cecil E. Stanley 
Commissioner of Education 
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Suggested Program Schedule for 
Training Sessions in Evaluation Skills 



Monday 




1:00- 4:00 


Using a Systematic Planning Model 


Tuesday 




9:00- 9:30 
9:30- 11:00 


Evaluation and Educational Decisions 
Part 1 of the Assessment Planning Chart 


Lunch 




12:00- 1:00 
1:00- 2:00 
2:00- 2:15 
2:15- 3:15 
3:15- 4:00 


Rationale and Description of a Student Information System 
Part II of the Assessment Planning Chart 
Coffee Break 

Discussion of the Assessment Planning Chart 
Trait Validation Procedures 


Wednesday 




9:00 • 12:00 


Indicators of Quality 



NOTEBOOK MATERIALS 
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OBJECTIVES: 



1. The participants should be able to state the goals for education by which their work 
should be influenced. 

2. They should be able to identify the functions they pursue which relate to those goals. 

3. The participants should be able to describe the four types of decisions necessary to 
educational development, and the kinds of evaluation which each require. 

4. They should be able to identify the kinds of decisions which are necessary for their 
functions. 

5. They should be able to name the kinds of evidence which are needed to judge their 
success. 

6. They should be able to relate the outcomes of their work to the goals for education 
which are influencing their work. 

7. The participants should be able to demonstrate several uses of the Student Information 
System. 

8. They should be able to apply the Systematic Planning Model to evaluation procedures. 

9. They should be able to identify and interpret the four Indicators of Quality. 
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PLANNING EDUCATIONAL EXPERIENCES THROUGH SYSTEMATIC PROCEDURES 



Carl Spencer 

Consultant, Milford Service Unit 



A. Rationale 

1. Professional education managers must be concerned with the ultimate purpose, 

effectiveness, and efficiency of the program. 

2. Specific answers to the following questions are needed to provide an effective and 

efficient program: 

a. What is the program to produce or achieve? 

b. How do you know the objectives are of importance for the program? 

c. How will one know when the program has achieved its ultimate purpose or 
goal? 

d. How will one measure the effectiveness and efficiency of the program in 
reaching its goal? 

e. How will one know when to revise or change the program design and goals? 

3. The professional educator's primary attention should be given to: 

a. The student/learner. 

b. His predictable learning success. 

c. His learning of skills, knowledges, and attitudes which are relevant to the role 
for which he is being prepared as a citizen, parent, producer, and consumer. 

4. Requirements for designing and implementing instructional programs and 

coordinating methods for managing these programs: 

a. Start with the learner. 

b. Design programs for instruction and management measured in terms of 

student learning or success. 

c. Establish measures of accountability for teachers, administrators, managers and 
learners which must be expressed in terms of specific performance, thereby 
allowing an objective evaluation of performance efficiency for all concerned. 

d. Provide for plans of action which assure the continuous production or 

development of relevant learner behaviors (skills, knowledges, and attitudes) 
assuring later success for the learner. 

e. Provide plans of action which efficiently and effectively integrate all members 

of the instructional and management teams while providing the greatest 

societal benefits at the least cost. 

f. End with the learner as the educational referent for planning, designing, 
implementing, evaluating, and managing of instructional and management 
programs. 
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5. Educators must establish more systematic procedures in order to satisfy this 

criterion of accountability to both learners and society. 

a. Producing predictable learner achievement requires a more objective and 
controlled process for: 

(1) Defining relevant skills, knowledges, and attitudes to be produced. 

(2) Defining the "learnings" to be produced in measurable terms in order to 
assess learner achievement. 

(3) Specifying in advance the performance requirements representing the 
ultimate learner achievements to be measured on completion of instruc- 
tion. 

b. Formal methods for planning, designing, implementing, evaluating and 
managing plans of action must be substituted for informal and less efficient 
methods. 

c. The formal method best suited to existing requirements in education and 
training is that of a system approach to educational problems. 



B. The Systematic Planning Model (James O'Hanlon and Carl Spencer) 

1. Any learning experience must be based on some concern felt by the learner. 

2. Establishing specific objectives 

a. What will the learner be able to do to demonstrate that he has reached the 
goal? 

b. The objectives should be closely related to one of the learner outcomes, stated 
as an observable behavior, and measurable. 

3. Assessing the learner's status 

4. Planning learning activities 

a. Set 

b. Modeling 

c. Appropriate practice 

d. Reinforcement 

5. Planning follow-up experiences 

a. Helping learners apply what they've learned. 

6. Planning evaluation procedures 

a. Did the learning succeed in getting the learner to incorporate the objective 
behavior in his work? 

b. What parts of the learning experience were most helpful to the learner? 
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OUTLINE OF MATERIALS 



I. What is Evaluation? 

A. Evaluation is a process providing some basic elements to administrators: 

1. An internal information system for better accomplishing the process of 
adaption in the course of change. 

2. Feedback — information which is intended to effect control. 

3. Information to decision-makers. 

B. Educational Evaluation is the process of obtaining and providing useful 
information for making educational decisions. 

1. Process — A specific, continuing activity which includes many methods and 
involves a number of steps or operations. 

2. Obtaining — Making available through processes (collecting, organizing, 
analyzing and reporting) and formal means (statistics, measurement, and 
computer processing). 

3. Providing — Fitting together into systems or subsystems that best sen/e the 
needs or purposes of evaluation. 

4. Useful — Appropriate to predetermined criteria evolved through the 
interaction of the evaluator and the client. 

5. Information — Descriptive or interpretive data about things that are real and 
their relationships. 

6. Educational Decisions - Choosing among alternatives which affect education. 

C. Evaluation is the process of providing and using information to make planning, 
structuring, implementing, and recycling decisions. 

II. Where Does Evaluation Fit in Educational Development? 

A. Evaluation may be one element within educational development. 

1. Educational needs 

a. Of the society 

b. Of the individual 
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2. Specify the problem 

a. Problem 1 — need 

b. This is not a problem that exists as an obstacle. 

3. Conceptualization of a strategy 

4. Implementing various types of strategies 

5. Developing an operational plan 

6. Developing processes, procedures, and materials 

7. Development of an evaluation design 

8. Testing the treatment 

9. Analysis, interpretation, and recommendation 

B. Evaluation may be a part of all the elements of educational development. 

1 . Planning decisions require context evaluation. 

a. These decisions specify changes needed in a program. 

b. This type of evaluation is aimed at continuous monitoring of the program 
to identify needs as they occur. 

c. The method of this evaluation is the definition and frequent measurement 
of variables which characterize the total program of interest. 

2. Structuring decisions require input evaluation. 

a. These decisions specify objectives; method, personnel, budget, schedule, 
organization, and content for use in effecting desired changes. 

b. Input evaluation is aimed at identifying and assessing relevant capabilities 
available to the program to define objectives, and to identify and assess 
alternative action plans for achieving the stated objectives. 

c. The method used in this evaluation is not specified. The prevalent 
practices, however, include panel deliberations, appeals to professional 
literature, and employment of consultants. 

3. Implementing decisions require process evaluation. 

a. These are the decisions used in carrying through the action plan. 

b. Process evaluation is aimed at detecting or predicting defects in the 
project operations and/or project design. 

c. The method used in this evaluation includes continuous identification and 
monitoring of the potential sources of failure in a project. 

4. Recycling decisions require product evaluation. 

a. These are the decisions used in determining the relation of outcomes to 
objectives and in determining whether to continue, terminate, evolve, or 
modify the activity. 




b. Product evaluation is aimed at relating outcomes to objectives and to 
context, input, and process. It is aimed at the measurement and 
interpretation of outcomes 

c. There are three basic methods used in this type of evaluation: 

(1) defining and measuring criteria associated with the objectives of the 
activity. 

(2) comparing these measurements with predetermined absolute or 
relative standards. 

(3) making interpretations of the outcomes, using the recorded context, 
input, the process information. 

5. Evaluation is a continuous process in this plan which may be initiated at the 
following times: 

a. prior to decision to write a proposal 

b. while writing a proposal 

c. during project implementation 

d. following or near the end of a complete cycle of the project 



III. Objectives of Evaluation 

A. Providing data related to system operation for administrative direction and control 
of change. 

B. Providing for the collection, organization and storage of data relative to programs 
and goals of the system. 

C. Providing for the analysis and evaluative procedures and techniques for program 
implementation, operation and termination. 

D. Providing a response capability for the facilitation of change and revision of 
programs and procedures. 

E. Providing for an ongoing evaluation of the evaluation system. 

F. Providing for the communication of information within and without the system. 



IV. Purposes of Statewide Evaluation 

A. To provide basic information for helping the student assess his own progress 
through the educational systems of the state, so he can understand himself, his 
educational needs, and his future possibilities. 

B. To provide the teachers and administrators in every school system with basic 
information for assessing the effectiveness of all principal phases of their 
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educational programs in sufficient detail to indicate specific steps to strengthen 
those programs. 



C. To provide the state education authority with basic information needed for 
allocating state funds and professional services to equalize educational 
opportunities for all children in the state. 

D. To provide lopal and state research agencies with data for generating and testing 
hypotheses cohceming the improvement of all aspects of the educational process. 

E. To provide every school system with strong incentives to experiment, under 
controlled conditions, with new and promising educational programs, materials, 
devices, and organizational arrangements. 

F. To periodically provide the state legislature and the general public with readily 
interpretable Information concerning the progress of the state system of education 
as a whole and of each local system. 

V. Criteria for Evaluating Proposed Project Evaluations 

(Note: The first four are general and common to any good evaluation information.) 

A. Internal Validity — information provided by the evaluation must show a reasonable 
correspondence to the phenomena which it purports to describe or interpret. It 
must have fidelity — be true. 

B. External Validity — the information must be generally transferable to similar 
situations beyond the one in which it was collected. 

C. Reliability - a repetition of the evaluation should produce essentially similar 
findings. 

D. Objectivity -- the data should be interpreted in much the same way by a number 
of people. 

Note: In addition to these first four, evaluative information requires: 

E. Relevance — the information must relate to the decisions to be made. 

F. Significance — the information must be weighted for its meaning in relation to the 
decision. Not all relevant information carries the same weight. 

G. Scope — the information must relate to all aspects involved in the decision. 

H. Credibility - the evaluator who secures information must be trusted by the 
decision-maker and those he must serve. 

12 
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I. Timeliness — the information must be prepared in time to be useful to the 
decision-maker. 

J. Pervasiveness — the information must get to all who need it. 
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“FROM EVALUATION THEORY INTO PRACTICE" (excerpt) 

Howard 0. Merriman 
Director of Evaluation and Research 
Columbus, Ohio, City Schools 



The evaluator's role in context evaluation may be largely one of delineating ihe 
information needs of decision-makers, establishing the criteria by which exceptions to the 
acceptable level of performance or behavior can be identified, developing the mechanisms 
for collecting this information, and providing the information to the appropriate 
decision-makers, decision influencers and audiences in such a way that it can be understood 
and utilized. Much information is available within school districts which could be of high 
utility in making decisions. However, access to such information in a timely manner may be 
restricted through a lack of knowledge of its availability or applicability, lack of 
data-handling capability or capacity, or through a lack of established networks for 

information flow. 

The evaluator works at two levels: (1) establishing a context evaluation information 
system and (2) facilitating the flow and use of such information. The former, establishing 
the system, is a means for the evaluator to employ the leverage principle, thereby 
multiplying his effectiveness and gaining efficiency. The evaluator's interest is in delineating 
information needs, and the establishing of a system to obtain and provide that information. 
He must utilize other roles in the organization to operate that system; for example, the 
teachers and counselors who administer tests which provide data on individual students and 
groups of students. The processing of test answer sheets by a data-processing unit is a part 
of obtaining information, just as the printouts of school summary data are a part of 
providing information. Neither of these operations needs to be maintained or operated by 
the evaluator. He is, however, interested in setting up the process or system whereby the 
data is obtained and provided. He will wish to test the information system for quality 
control, since the reliability, validity and precision of the information provided through that 
system are very important to him. 

The evaluator, while facilitating the flow of useful information must, at the same time, 
avoid overburdening the decision-maker with more information than he can u^. He may 
establish methods of exception reporting, for example, through which the decision-maker 
receives information only when previously established upper and/or lower limits have been 

exceeded. 

Once such a context evaluation information system is established, a major role of the 
evaluator may be that of resetting criterion levels or establishing new criterion measures 
which will reflect new conditions of concern to the educational community. 

In summary, an evaluation department which is to provide information for planning 
decisions must include a component which has as its mission the developnient of an 
information system by: delineating, with decision-makers, their information needs; 



determining criteria and levels to be used in analyzing information acquired and establishing 
mechanisms to obtain this information; and building networks to insure the flow of such 
Information to the appropriate decision-maker(s). Ideally, over the long run, once such a 
system is established, the evaluator may only have to exercise a quality control function. 
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STUDENT INFORMATION SYSTEM 






The gathering of student information, keeping it updated, and using it in a meaningful 
way to assist students in learning has challenged educators for many years. The Student 
Information System (SIS) is an effort to meet the challenge in the computer age. In the 
past we have been forced to look only at those learning objectives for which we had 
measures — IQ and achievement tests. But now we are starting to realize, in fact being 
forced to realize, that we have a number of objectives, not just scores on tests. In 

behavioral research, learner differences must be identified, as well as attitude highs, IQ 

levels, and all the other aspects of the student. The objectives that are used, must all be 
Identified, and program ties used to attain those objectives must be determined. Then its 

not a matter of one school against another, or one program against another program, but 

rather which kind of things work for which students and under which conditions. Then it's 
not a matter of whether a teacher is "good" or "bad," but rather, what specific skills he 
has that are effective and for which kinds of students. 

The first step was to search the research literature and select the variables and 
Information readily available in the schools which correlate in any way with learner and 
program objectives at various levels. Personnel with skills in system analysis, data processing, 
school administration, teaching, learning theory, psychology, social work, occupational 
guidance, personality theory, educational measurement, curriculum, and so on, all par- 
ticipated in the development of SIS. The second step was to determine the best sources of 
the needed information. They are the teacher, parents, students, the various specialists, and 
current school records. After the content was outlined and the sources described, 
instruments (check lists, questionnaires, etc.) were developed for field trials for gathering 
and analyzing data. 

The final step was to determine the best ways to analyze the data so that it would be 
meaningful and useful, and to organize distribution techniques to teachers, administrators, 
etc. A method was devised whereby the instruments can be scored either by hand or with 
an electronic optical scanner for computer analysis. The computer output of SIS includes 
administrative reports as well as records, evaluations, and diagnostics of individual students 
or any particular group of students. 

The primary purpose of SIS is to provide schools with the necessary materials and 
step-by-step procedures that will: 

1. Help determine each student's individual learning and program needs. 

2. Provide teachers, administrators, parents, and students with descriptions of these 
needs. 

3. Provide reports which aid in specifying program planning to meet student needs. 
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4. Provide baseline information to evaluate attainment of learning and program 
objectives. 

5. Provide evaluation data on the effects of any project or program on different kinds 
of student performances for any student group. 

6. Provide necessary data for administrative reports and for program planning. 

7. Provide easily accessible and easily interpretable, permanent records for specific 
target groups of students, comparison of test scores, expected percentages of 
students needing special education, demographic analysis, etc., for long-range 
planning. 

8. Identify incipient learning problems and developmental difficulties at preschool 
and kindergarten levels. 

9. Provide the basis for an analysis of agreement/disagreement between 
parent/school/student expectations of the educational system (a major 
administrative problem in many districts.) 

Information for SIS is provided by a set of data forms which gather information from 
students, parents, teachers, and specialists. These instruments contain information similar to 
the current Cumulative Record, and therefore, could eventually replace it. The basic forms 
are: 

1. Parent Questionnaire — The parent describes the environment, interests, abilities, 
and characteristics of his child. 

2. Student Check Lists - There are three check lists through which the teacher 
describes the achievement, problems, and behavioral characteristics of the student. 

3. Student Questionnaire — The student describes his interests, attitudes, plans, and 
so on. Three levels are available — elementary, secondary, and post-high school. 

4. Sfjeciai Forms — For gathering test scores, administrative information, specialists' 
report, and the like. 

5. Other Forms — May be developed for individual district requirements and added to 
the system to meet specific district needs. 





REPRESENTATIVE LEARNING OUTCOMES AND POSSIBLE 
METHODS OF EVALUATION 



LEARNING OUTCOMES 


METHODS OF EVALUATION 


A. Application 

Concept Acquisition 
Memorization of Facts 
Problem Solving 
Reading Comprehension 
Skills (number, etc.) 


A. Objective Test 

Product Evaluation 
Rating Scale 
Checklist 


B. Performance 


B. Rating Scale 
Checklist 

Product Evaluation 


C. Classroom Behavior 


C. Rating Scale 
Checklist 

Attendance Record, etc. 


D. Interest 


D. Questionnaire 
Checklist 

Interest Inventory 
Factual Vocabulary Test 


E. Attitude 


E. Rating Scale 
Questionnaire 
Checklist 
Objective Test 


F. Aspiration Level 


F. Rating Scale 
1 nterview 

Simple Objective Test 
Word Association Test 
Open Ended Sentences 


G. Adjustment 


G. Rating Scale 

Anecdotal Report 
Interview, Sociogram 



The methods of evaluation would be simplified and the process facilitated if one instrument 
were designed to coordinate all the above methods. 
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instructions for Assessment Planning Chart 



Given the general concerns in education that the people of Nebraska seek for their 
children, department members will decide which goals are appropriate to their positions. 

Working in groups related by common concerns, participants will write functions of 
their work that relate to that goal. The functions will be placed under the kind of decision 
(planning, structuring, implementing, or recycling) which the function exhibits. Participants 
will then be asked to write down what kind of evidence is needed to show if they have 
been successful in that function. For each function, which involves changes in pupil 
behaviors, participants should list the desired behavior change. Each behavior should be 
listed separately and should describe specifically what the pupil should do at the conclusion 
of the project. 

Traditionally, one or more of the methods or instruments of evaluation listed on the 
previous page could be chosen to evaluate the presence of the desired behavior change. But 
assuming that we have one instrument which could be used to evaluate such changes (the 
Student Information System), then participants will be asked to choose items on the 
Confidential Student Questionnaire (Level II) Part II which they feel would measure this 
behavior change. The numbers of the chosen items should be recorded under the final 
column in the rquare joined to the behavior which it tends to measure, along with the 
numbers of items from the Student Behavior Check List which participants feel also 
measure that trait. 
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ASSESSMENT PLANNING CHART (Part I) 
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(Part II) ITEMS OF EVALUATION 

BEHAVIOR CHANGES SQ II | 5gCL 



TRAIT VALIDATION PROCEDURES 



Before any scale which attempts to identify creative students can be developed, the 
goals and objectives of the programs developed for such students must be determined. These 
goals and objectives must be continually referred to in the consideration of both students 
and scale items. It is also important that people involved in the selection of scale items 
understand the significance of each item. It is difficult to indicate that most items relate to 
a specific attitude or behavior. Rather, many items are included because different groups of 
students respond to them with different kinds of answers. If one item reads "I hate dogs," 
the important consideration is not whether an individual likes or dislikes dogs. Rather, the 
usefulness of the item lies in the tendency of certain types of students to give either a 
"high" or "low" response to it. This is known as the "discriminating power" of the item. 

There are two basic steps in developing a scale for the identification of those students 
who might benefit from a program designed to help develop creative potential. In the first 
step, half of those people who are responsible for identifying such students actually fill out 
the Student Questionnaire Level II (SQ II) themselves. They respond to each item as they 
feel a student who possesses creative potential would. The items which are commonly 
responded to as "high" or "low" by this group, then become those items which a creativity 
scale might contain. 

The other half of this group should respond to each item on the SQ II as they feel a 
student of very limited creative potential would. The items which they agree such a student 
would make a "high" or "low" response on may also become components of a creativity 
scale. Both halves of the group should then review each item on theSQ II to determine the 
degree to which it might relate to the goals and objectives of the program for which 
potential students are being selected. 

In the second step, teachers or other individuals in a position to make student 
evaluations identify a group of students who most nearly exhibit the behaviors associated 
with mastery of the objectives of a creativity program. They are also asked to identify a 
group of students who seem to have very little creative potential. The answers these two 
groups of students made on the SQ II are then analyzed to find on which items their 
responses were consistently different. Each item on which these groups consistently differed 
in their responses is a potential part of the creativity scale. In this process it is very 
important that those people who identify individual students limit their consideration to 
indicators of creative potential alone. Other factors of the student's personality or 
background (personal habits, appearance, etc.) must be given absolutely no weight. 
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ANNOTATED BIBLIOGRAPHY 



Beatty, Walcott H., Chairman and editor, Association for Supervision and Curriculum 
Development, NEA, Improving Educational Assessment and an Inventory of Measures 
of Affective Behavior, ASCD, 1969. 

The first part of this book is a collection of four papers which analyze the system we 
have for collecting and using data and which propose extensions of the system to catch new 
purposes and new dimensions. An important chapter by Daniel L. Stufflebeam discusses 
evaluation's role in decision making. The last part is an annotated resource list to enable us 
to measure affective behavior. 



Bush, Donald 0., and Willard G. Jones, "Educational Development - from research to 
practice. . .," Rocky Mountain Educational Laboratory, Inc., Greeley, Colorado, April, 
1969. 

This document discusses the nature and process of educational development. One of 
the most beneficial aspects of the booklet is a presentation portraying a model for this 
development with very precise definitions. It locates evaluation as a necessary element of 
the process being both dependent on previous elements and depended upon. 



Davis, Joseph L., and Martin W. Essex, Educational Evaluation, Columbus, Ohio, 1969. 

This book is a collection of speeches given at a conference sponsored by the Ohio 
Department of Education. They deal with the need and importance of accurate measunng, 
appraising and assessing instruments, as well as evaluation designs and models and the use of 
evaluation evidence. 



Dyer, Henry S., "Statewide Evaluation - What Are the Priorities," Phi Delta Kappan, June, 
1970. 

This article does an excellent job of stating the purposes of statewide evaluation. These- 
purposes are then ranked by the author in order as he sees their importance with individual 
justification of those ratings. 

Foley, Walter J., "The Future of Administration and Educational Evaluation," Educational 
Technology, July, 1970. 

This article discusses the change concept of educational systems. It differentiates 
between the administrative function (managing change) and the role of evaluation (providing 
information). With this difference in mind he states the objectives of evaluation and designs 
a model evaluation system. 
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Hemphill, John K., "Educational Development," draft, February, 1969. 

This article distinguishes between two types of educational development: the product 
development approach and the change support approach. The emphasis of change support 
approach is on the continuation of educational development as a process. It is never to be 
completed since it can never be said to reach a point where further improvement is not 
possible. Objectives then become temporary states and evaluation becomes feedback. 



Merriman, Howard 0., "From Evaluation Theory Into Practice," Journal of Research and 
Development in Education, Vol. 3, No. 4, Athens, Georgia, Summer, 1970. 

This article emphasizes the four kinds of decisions made in educational development 
and the kinds of evaluation they require. Each type of evaluation is then described in 
relationship to the information it will produce in order to arrive at one of the four kinds of 
decisions. 



Olson, Thomas A., and Lelia Marvin, "Evaluation: One State's Approach," American 
Education, Vol. 6, No. 4, May, 1970. 

This article describes the process used by the Department of Education in Illinois to 
evaluate their 55 Title III projects. Their major attempt was described as "humanizing 
evaluation through consultants." Contrary to most evaluation techniques, they were 
encouraged to make value judgments and offer personal opinions. 



Provus, Malcolm, "Toward a State System of Evaluation," Journal of Research and 
Development in Education, Vol. 3, No. 4, Athens, Georgia, Summer, 1970. 

This article discusses the cycle of an evaluation system and the areas deserving special 
emphasis. The whole system is based on the idea that one plans on the basis of knowledge 
derived from evaluating one's successes and failures. The author also gives a list of questions 
which must be answered to make the system work. 



Spencer, Carl R., Planning Workshops Through Systematic Procedures, pamphlet, ASCD 
25th Annual Conference, March 16, 1970. 

This article provides a rationale for using systematic procedures in planning educational 
experiences. The pamphlet then follows up the rationale with the actual model for 
systematic planning. This model provides for evaluation procedures when it establishes 
objectives. 



Stufflebeam, Daniel, Robert Hammond, and Egon G. Guba, "Planning and Implementing 
Title III Evaluations," United States Office of Education, April, 1968. 

This statement begins with a discussion of the variables of the three dimensions of 
evaluation— instruction, institution, and behavior. They define evaluation in terms of 



providing information for four types of educational decisions. They describe the kinds of 
evaluation necessary for each of these decisions in terms of objectives and methods. 



Wainwright, Bruce, Manual of Student Information System, Lincoln, Nebraska, April, 1970. 

This manual explains the Student Information System (SIS), its purpose, how it was 
developed, its structure and content, and all possible uses. It includes information on 
administering the questionnaires, as well as scoring methods and interpretation. 



New England Educational Assessment Project, A Guide to Assessment and Evaluation 
Procedures, Providence, Rhode Island, February, 1967. 

This guide is aimed at aiding local school systems in the complex work of evaluation of 
Title I and other special education projects. It provides the general design of an evaluation 
program and strongly suggests that the instruments used for measurement are pertinent to 
the project activity and to the pupils involved. 



Office of the Utah State Board of Education, Handbook of Student Information System, 
Salt Lake City, Utah, February, 1970. 

This handbook accompanies the SIS manual and permits a person without special 
training to utilize SIS data in a variety of ways. Also included in this manual are several 
examples of research designs which are appropriate for both individual and group 
application of SIS derived data. 



United States Office of Health, Education, and Welfare, Preparing Evaluation Reports — A 
Guide for Authors, United States Government, Printing Office, Washington, 1970. 

This guide is to aid evaluators in preparing reports through decisions of what to include 
and how to report it. It follows a very logically developed pattern consisting of the context 
of the program, the program itself, evidences of changes, and recommendations. 
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APPENDIX 



ERLC 






INDICATORS OF QUALITY 
adapted from a brochure printed by 

The Institute of Administrative Research 
Teachers College — Columbia University 



The Indicators of Quality is a new instrument for measuring school quality by 
observing the behavior that goes on in the classroom. It is based on characteristics of 
school behavior which are thought to be basic to quality. Four basic ones have been 
identified: individualization, interpersonal regard, creativity, and group activity. These were 
determined by educational experts who were asked to decide upon what bases they would 
judge school quality. They may be observed in the teaching-learning procedure. 

Certain key concepts were found to define the content of the four criteria. They are: 

Individualization 

1. Knowledge of pupils 

2. Physical facilities 

3. Different tasks 

4. Participation 

5. Communication 

6. Modification of questioning 

7. Complementary teacher-pupil roles 

8. Time for growth 

9. Individual evaluation 

Interpersonal Regard 

10. Demeanor 

11. Patience 

12. Pupil involvement 

13. Physical movement 

14. Respect 

15. Error behavior 

16. Pupil problems 

17. Atmosphere of agreement 

18. Teacher-pupil identification 

19. Evaluation as encouragement 

Creativity 

20. Time for thinking 

21. Abundance of materials 

22. Skills for thinking 

23. Testing ideas 

24. Unusual ideas 

25. Question and answer technique 
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26. Self-initiated activity 

27. Opportunity for speculation 

28. Evaluation as motivation 

Group Activity 

29. Physical arrangement 

30. Teacher purpose 

31. Decision-making 

32. Intercommunication 

33. Conflict resolution 

34. Cooperation 

35. Role distribution 

36. Group goals 

37. Group personality 

38. Consensus 

39. Group evaluation 

40. Teacher's group role 

The instrument includes 51 items based on these 40 key concepts. Some of the 
concepts relate exclusively to teacher behavior, or pupil behavior; others may be discerned 
both in what teachers do and what pupils do. 

With a criterion of quality exhibited in this instrument, objective scores obtained can 
be statistically related to other measures that result from administrative policy: level of 
financing, teachers' salaries, teacher characteristics, staff deployment and class size, com- 
munity conditions, pupil background characteristics, pupil achievement, state legal 
regulations and a host of other factors that, in one way or another, are presumed to 
influence the quality of education. A study of these factors should indicate what changes 
could be made in the way our education is supported and managed. 
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IN FORMATION 



O 

ERLC 



COLLECTION INSTRUMENTS 






\ 

► 

\ 



Name 



In anticipation of the training sessions in evaluation (November 23, from 1:00-4:00, 
November 24, from 9:00-4:00, and November 25, from 9:00-12:00) we are asking you to 
complete this questionnaire. You will again be grouped on the basis of your choices. Please 
return this sheet to Vickey Radcliffe, 411 Building, South 13th Street, Lincoln. 



a 



O 






O 

ERIC 



Below is a list of educational concerns of the people of Nebraska, which might be 
considered as goal statements. Using a scale from 1-7, put a check in the box below the 
number which indicates how concerned your work should be with what the statement 
suggests. A check under #1 would mean that you are directly concerned with the 
statement, while a check under #7 would show no concern at all. 



1 


2 


3 


4 


5 


6 


7 

















1. To aid in the development of materials and programs which will help 
students prepare themselves for problems they might face in our contemporary society (Ex. 
drugs, sex education, decision making). 



1 


2 


3 


4 


5 


6 


7 

















2. To initiate surveys to determine the number and description of physically 
or mentally handicapped persons in Nebraska, so that program development and resource 
allocation can be adjusted to their needs. 



1 


2 


3 


4 


5 


6 


7 

















3. To aid local schools in developing programs to help each student select 



and prepare himself for the world of work. 



1 


2 


3 


4 


5 


6 


7 

















4. To help schools identify their students as individuals, so that guidance. 



academic, and vocational programs are more relevant to their needs. 



1 


2 


3 


4 


5 


6 


7 

















5. To undertake studies to determine what "teaching competency" consists 
of and develop a system by which teachers, students and administrators can cooperate in 
the measuring of "teacher competency". 



1 


2 


3 


4 


5 


6 


7 

















6. Sup/ey leisure activities available in Nebraska and develop suggested 



physical education programs which would prepare students for those activities. 



1 


2 


3 


4 


5 


6 


7 

















7. To examine planning services that the State Department of Education 



makes available to those people who are considering reorganization of their districts. 



1 


2 


3 


4 


5 


6 


7 

















8. To help schools provide programs and services that will strengthen their 



local communities economically and socially. 
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Name 



FINAL EVALUATION OF TRAINING SESSIONS 
IN EVALUATION SKILLS 



Part I: 

1. Put a (+) in front of the topic which you believe will be most useful to you in 
your work. 

2. Put a (0) in front of the topic which you believe will be least useful in your work. 

3. Put a (✓) in front of the topic which you feel needed more or better coverage. 

4. Put a (X) in front of the topic which you feel needed less coverage. 

Using a Systematic Planning Model 

Evaluation and Educational Decisions 

Information Systems and Evaluation 

Indicators of Quality 

1. Put a (+) in front of the methods which you feel were most effectively used. 

2. Put a (✓) in front of the methods which were adequate. 

3. Put a (0) in front of the methods which were least effectively used. 

Media 

Notebooks 

Group discussions 

Department speakers 

Outside speakers 




00 

o 

o 

o 



Part II; 

1. What problems do you anticipate in using what you have learned? 



2. What contribution were you able to make in these sessions? 



3. What was personally most meaningful to you about these sessions? Why? 



4. If anything happened which aroused strong feelings in you (pleasant or 
unpleasant), briefly describe it. 



S 

H 
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Name 



QUESTIONNAIRE FOR EVALUATION SKILLS 



Part I: 

In the blanks preceding each statement of function, code the kind of decision and 
evaluation which that function exhibits in educational development. 





Decisions 


Evaluation 




Code 


Code 






A. Planning 


1. 


Product 




B. Structuring 


2. 


Process 




C. Implementing 


3. 


Input 




D. Recycling 


4. 


Context 


Decisions 


Evaluation Functions 







Seeking information from parents, students, visitors and 
teachers about the impact of a Title III project and comparing 
their responses with the project's objectives. 

Seeking information from patrons and educators about the 
problems in education which they believe face the people of 
Nebraska. 

Monitoring the reading progress of students involved in a 
project where the main objective is to innprove their reading 
level by 2 grades. 

Comparing the proportion of students retained in school 
between the 11th and 12th grades before and after the 
employment of a guidance counselor. 

Having a representative group study literature on 
differentiated staffing, poll community attitude towards it, 
compare facilities available with those necessaiy to initiate it, 
and survey staff attitudes and availability. 

Identifying the factors behind student unrest. 

Measuring the learning gain from a television series. 

Conducting a site visit to an ESEA Title ill project. 

Prior to initiating a humanities course, surveying available 
literature and noaterials, studying and setting course objectives 
by the English Department and deciding on interested, 
capable teachers for conducting the class. 

Interviewing parents of children involved in classes for the 
handicapped to get their opinions about program success. 
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Part II: 



Below is a list of situations in a classroom. Identify the concept which the content suggests. 

Put the abbreviation of one of the four Indicators of Quality in front of each situation: 

1= Individualization 
IR= Interpersonal Regard 
C = Creativity 
GA= Group Activity 

1 . Pupils have the stimulation of materials and other resources in great richness 

and variety. 

2. Pupils are cooperating in putting on a play. 

3. Pupils are seated in a circle. 

4 . A pupil giving a lengthy, detailed answer is listened to and accepted by the 

class and teacher. 

5 . Pupils are given time to choose books to read, while the teacher helps 

individual students pick books they are interested in and capable of reading. 

Part III: 

Put a T in front of the statements you believe to be true and an F in front of those which 

you believe are false. Use the back to explain any answers to statements which you feel are 

not clearly stated. 

1. To operationalize evaluation, we need objectives. 

2. There is little need to make provisions for evaluation procedures when 

planning learning activities. 

3. Some kind of evaluation is needed prior to the decision to write a proposal. 

4. An evaluation department which is to provide information for planning 

decisions must include e component which has as its mission the development 
of an information system. 

5 . The Indicators of Quality can be effectively used to evaluate the performance 

of teachers. 



